Method of and scaling unit for scaling a three-dimensional model and display apparatus

ABSTRACT

A method of scaling a three-dimensional model ( 100 ) into a scaled three-dimensional model ( 100 ) in a dimension which is related with depth which method is based on properties of human visual perception. The method is based on discrimination or distinguishing between relevant parts of the information represented by the three-dimensional model for which the human visual perception is sensitive and in irrelevant parts of the information represented by the three-dimensional model for which the human visual perception is insensitive. Properties of the human visual perception are e.g. sensitivity to a discontinuity in a signal representing depth and sensitivity to a difference of luminance values between neighboring pixels of a two-dimensional view of the three-dimensional model.

The invention relates to a method of scaling a three-dimensional modelinto a scaled three-dimensional model in a dimension which correspondsto a viewing direction of a viewer towards the three-dimensional model.

The invention further relates to a scaling unit for scaling athree-dimensional model into a scaled three-dimensional model in adimension which corresponds to a viewing direction of a viewer towardsthe three-dimensional model.

The invention further relates to an image display apparatus comprising:

-   -   receiving means for receiving a signal representing a        three-dimensional model;    -   a scaling unit for scaling the three-dimensional model into a        scaled three-dimensional model in a dimension which corresponds        to a viewing direction of a viewer towards the three-dimensional        model; and    -   display means for visualizing a view of the scaled        three-dimensional model.

The probability that the size of a three-dimensional scene does notmatch with the display capabilities of an image display apparatus ishigh. Hence, a scaling operation is eminent. Other reasons why scalingmight be required is to adapt the geometry of the three-dimensionalmodel representing the three-dimensional scene to a transmission channelor to adapt the three-dimensional model to the viewer's preferences.

Linear scaling operations on a three-dimensional model representing athree-dimensional scene are well known. An embodiment of the imagedisplay apparatus of the kind described in the opening paragraph isknown from the U.S. Pat. No. 6,313,866. This image display apparatuscomprises a circuit for acquiring a depth information maximum value froma first image signal. The image display apparatus further comprises aparallax control circuit to control the amount of parallax of a secondimage signal on the basis of depth information contained in the firstand second image signals such that an image corresponding to the secondimage signal can be three-dimensionally displayed in front of an imagecorresponding to the first image signal. A three-dimensional imagesynthesizer synthesizes the first and second image signals which havebeen controlled by the parallax control circuit, on the basis of theparallax amount of each image signal, such that images correspond tothat first and second image signals in the three-dimensional displayspace. Scaling of depth information is in principle performed by meansof a linear adaptation of the depth information except for depthinformation which exceeds the limits of the display capabilities. Theselatter values are clipped.

A disadvantage of depth adaptation or scaling is that it might result inreduction of depth impression. Especially the linear depth scaling mightbe disadvantageous for the depth impression of the scaledthree-dimensional model.

It is an object of the invention to provide a method of the kinddescribed in the opening paragraph which results in a scaledthree-dimensional model which resembles the original three-dimensionalmodel perceptually and which has a pleasant three-dimensionalimpression.

The object of the invention is achieved in that the method is based onproperties of human visual perception of the viewer. These propertiesmight be a.o.:

-   -   sensitivity to a discontinuity in the three-dimensional model in        a dimension which is related with depth, i.e. in a signal        representing depth; Notice that the dimension which corresponds        to a viewing direction of a viewer towards the three-dimensional        model equals depth.    -   sensitivity to a difference of luminance values between        neighboring pixels of a two-dimensional view of the        three-dimensional model, i.e. amount of texture;    -   sensitivity to color values of pixels of a two-dimensional view        of the three-dimensional model; and    -   sensitivity to a difference of particular color values between        neighboring pixels of the two-dimensional view of the        three-dimensional model.        The method according to the invention is based on discrimination        or distinguishing between relevant parts of the information        represented by the three-dimensional model for which the human        visual perception is sensitive and in irrelevant parts of the        information represented by the three-dimensional model for which        the human visual perception is insensitive. The relevant parts        should be stressed when the three-dimensional model is scaled,        optionally causing an impairment or even deformation of the        irrelevant parts.

An embodiment of the method according to the invention comprises adiscontinuity detection step to detect a discontinuity in thethree-dimensional model in the dimension which is related with depth. Anaspect of linear scaling is that the geometry of the three-dimensionalmodel can be maintained. But this is not a strong requirement for depthscaling, because humans are not very sensitive to adaptation of theamount of depth. The best proof for this is the fact that humansappreciate normal two-dimensional video which is entirely “flat”. Thisphenomena is also discussed in the article “Just enough reality:Comfortable 3-D viewing via microstereopsis”, by M. Siegel et al., inIEEE Transactions on Circuits and Systems for Video Technology, vol. 10,no. 3 pp. 387-396, 2000. Hence, the limited depth offered by e.g. linearscaled three-dimensional models still gives a three-dimensionalimpression. Nevertheless, humans do very well notice that in that casethe three-dimensional impression is small. This is not due to the factthat absolute depth values of the scaled three-dimensional model aresmall, but due to the fact that the depth discontinuities. are small. Inother words, linear depth scaling affects the size of the depthdiscontinuities, resulting in a reduced depth impression. In general,humans are very sensitive to topology of a scene and especiallysensitive to depth discontinuities but less sensitive to geometry.Humans, very well observe that objects are in front of each other ande.g. partly occlude each other. However absolute depth, that means theactual distance between objects, is of less importance. This impliesthat even with a limited range of depth values, still a pleasantthree-dimensional impression can be made as long as the topology ispreserved and hence the depth discontinuities are maintained or evenamplified.

Another embodiment of the method according to the invention comprises:

-   -   a luminance contrast detection step to determine a particular        luminance contrast value of a particular pixel with a        neighboring pixel, with the particular pixel belonging to a        two-dimensional image which is a view of the three-dimensional        model; and    -   a luminance contrast dependent scaling step to scale a depth        value of an element which corresponds with the particular pixel        on basis of the particular luminance contrast value.        The theory behind this embodiment will be explained by means of        an example. If a white object is located in front of a white        background, e.g. wall, it is hardly visible. This means that        scaling depth of the scene with this white object and the white        background will not substantially influence the        three-dimensional impression. If there is a second, e.g. black        object in the scene, then the depth scaling of the        three-dimensional model of this scene should be controlled by        the difference of depth between the black object and the wall.        The difference in depth between the white object and the wall is        not very significant for the scaling and should not or hardly be        taken into account for the depth adaptation.

An embodiment of the method according to the invention comprises:

-   -   a range detection step to estimate a range of depth values in a        portion of the three-dimensional model in the dimension which is        related with depth; and    -   a comparison step to compare the range of depth values with an        output range of depth values.        In general, scaling is a mapping of information from an input        domain to an output domain. If the ranges of the input and        output values are known, the appropriate scaling can be        determined. In most cases, the range of output values is known,        because this range corresponds with the display capabilities of        the display apparatus. However if the range of input values is        unknown then this range should be determined. The advantage is        that an optimal scaling can be achieved.

Modifications of the method and variations thereof may correspond tomodifications and variations thereof of the scaling unit and of theimage display apparatus described.

These and other aspects of the method, of the scaling unit and of theimage display apparatus according to the invention will become apparentfrom and will be elucidated with respect to the implementations andembodiments described hereinafter and with reference to the accompanyingdrawings, wherein:

FIG. 1A schematically shows a depth profile of an originalthree-dimensional model;

FIG. 1B schematically shows a depth profile of a linearly scaledthree-dimensional model;

FIG. 1C schematically shows a depth profile of a three-dimensional modelscaled with the method according to the invention;

FIG. 2A schematically shows an embodiment of a scaling unit based ondiscontinuity preservation;

FIG. 2B schematically shows an embodiment of a scaling unit based ondiscontinuity preservation comprising a low-pass filter;

FIG. 2C schematically shows an embodiment of a scaling unit comprising aclipping unit;

FIG. 3 schematically shows an embodiment of a scaling unit based onluminance contrast detection; and

FIG. 4 schematically shows an embodiment of a three-dimensional displayapparatus.

Corresponding reference numerals have the same meaning in all of theFigs.

There are several types of:

-   -   methods of and equipment for the acquisition or generation of        three-dimensional information;    -   three-dimensional models for the storage of three-dimensional        information;    -   conversions of data represented by one type of three-dimensional        model into another three-dimensional model; and    -   image display apparatus for the visualization of        three-dimensional information.

First, some types of three-dimensional models will be described briefly.

-   -   Wireframes, e.g. as specified for VRML. These models comprise a        structure of lines and faces.    -   Volumetric data-structures or voxel maps (Voxel means volume        element). These volumetric data-structures comprise a        three-dimensional array of elements. Each element has three        dimensions and represents a value of a property. E.g. CT        (Computer tomography) data is stored as a volumetric        data-structure in which each element corresponds to a respective        Hounsfield value.    -   Two-dimensional image with depth map, e.g. a two-dimensional        image with RGBZ values. This means that each pixel comprises a        luminance, a color and a depth value.    -   Image based models, e.g. stereo image pairs or multiview images.        These types of images are also called light fields.

Data represented with a wireframe or a two-dimensional image with depthmap can be converted by means of rendering into data represented with avolumetric data-structure or image based model.

The amount of depth which can be realized with a three-dimensional imagedisplay apparatus depends on its type:

-   -   With a volumetric display device the amount of depth is fully        determined by the dimensions of the display device.    -   Stereo displays with e.g. glasses have a soft limit for the        amount of depth which depends on the observer. Observers might        become fatigued if the amount of depth is too much caused by a        “conflict” between lens accommodation and mutual eye        convergence.    -   Autostereoscopic display devices, e.g. an LCD with a lenticular        screen for multiple views have a theoretical maximum depth value        which is determined by the amount of views. This maximum depth        value can be exceeded resulting in loss of sharpness. There is a        relation between the type of three-dimensional image display        apparatus and the appropriate type of three-dimensional model in        which the three-dimensional information should be provided.

FIG. 1A schematically shows a depth profile 100 of an originalthree-dimensional model. FIG. 1B schematically shows a depth profile 102of the corresponding linearly scaled three-dimensional model. FIG. 1Cschematically shows a depth profile 108 of a three-dimensional modelscaled with the method according to the invention. In FIG. 1B and FIG.1C a top view of a three-dimensional image display apparatus 104 isshown. The gray box indicates the depth range 106 which is applicable tothis three-dimensional image display apparatus 104. This box resemblesthe display capabilities of the three-dimensional image displayapparatus 104 in the dimension related with depth.

First, depth profile 100 is compared with depth profile 102. It can beseen that the continuous portions 101-107 are mapped to the respectivecontinuous portions 115-121. Their shapes are not modified. That meansthat elements belonging to a particular continuous portion, e.g. 115have equal depth values. The C₀-discontinuities 109-113 are mapped tothe C₀-discontinuities 123-127. The sizes of the C₀-discontinuities123-127 are smaller than the sizes of the C₀-discontinuities 109-113.Thus, the depth impression is reduced.

Next, depth profile 100 is compared with depth profile 108. It can beseen that the continuous portions 101-107 are mapped to the continuousportions 129-135. Their shapes are modified. That means that elementsbelonging to a particular continuous portion, e.g. 129 do not have equaldepth values although these elements did have equal depth values beforescaling. In fact, parallel surfaces in the original scene now haveslanted orientations. The C₀-discontinuities 109-113 are mapped to theC₀-discontinuities 137-141. The sizes of the C₀-discontinuities 137-141are larger than the sizes of the C₀-discontinuities 109-113. Note thatenlargement of these sizes is not required. Although the total depthrange is reduced the depth impression is increased. This is achieved bystressing the C₀-discontinuities 109-113. The continuous portions101-107 are deformed on behalf of the C₀-discontinuities 109-113.Because humans are not very sensitive for absolute depth values thesedeformations are hardly perceived. And if perceived then thesedeformations are not annoying.

FIG. 2A schematically shows an embodiment of a scaling unit 200 based ondiscontinuity preservation. Especially C₀-discontinuities are ofinterest. The scaling unit 200 comprises:

-   -   a high-pass filter 202 arranged to filter a signal corresponding        to depth in order to detect discontinuities;    -   an automatic gain controller 204 based on peak-detection or        optionally on envelop detection to determine the input range of        depth values; and    -   a normalization unit 206 to adapt the filtered signal to the        output range based on the input range detected by the automatic        gain controller 204. At the input connector 208 a depth signal        is provided and the scaling unit 200 provides a scaled depth        signal at its output connector 210. The high-pass filter 202 is        arranged to discriminate between relevant portions and        irrelevant portions of the signal, i.e. discontinuities and        continuous portions respectively. The filtered signal is        normalized by means of the normalization unit 206 based on a        local maximum of the filtered signal or based on a value which        is a calculated by means of a “walking average” of maximum        values. The working of the scaling unit 200 as depicted in FIG.        2A is substantially equal to the working of scaling unit 201 as        depicted in FIG. 2B and will be described in connection with        FIG. 2B.

FIG. 2B schematically shows an embodiment of a scaling unit 201 based ondiscontinuity preservation comprising a low-pass filter 212 and asubtraction unit 214. In stead of using a high-pass filter 202 it ispossible to apply a low-pass filter 212 in combination with asubtraction unit 214. By means of subtracting a low-pass filtered signalfrom an original signal the high frequency components are maintained.This embodiment of a scaling unit 201 is based on such a filteringapproach. Next follows a mathematical description of this embodiment.The depth of the original scene is D₀ (x, y) where x and y are imagecoordinates. D₀ (x, y) can be expressed in any unit related to depth,e.g. pixel-disparity or meters. Let the unit be pixel-disparity. Thescaled depth is D_(c) (x, y) and the depth range of thethree-dimensional image display apparatus on which the information willbe visualized is given by −k<D_(C)<k. Notice that most three-dimensionalimage display apparatus have a symmetric limitation around zero depth.If this is not the case, the symmetric k limitation can be applied incombination with an addition unit which is arranged to add apre-determined constant to the scaled signal. A typical value for k inthe case of a autostereoscopic display with a lenticular screen with 9views is 4. This follows from k=(9−1)/2. The scaling unit 201 can bemathematically described with: $\begin{matrix}{D_{c} = {k\frac{D_{o} - {F_{\sigma_{1}}\left( D_{o} \right)}}{2{F_{\sigma_{2}}\left( {{D_{o} - {F_{\sigma_{1}}\left( D_{o} \right)}}} \right)}}}} & (1)\end{matrix}$with F_(σ1) and F_(σ2) low pass filters, e.g. Gaussian filters withvariance parameters σ₁ and σ₂ equal to 50. However the type of filter orits parameters can be varied extensively. The variance parameters mightbe selected such that an entire depth map belonging to a completetwo-dimensional image is covered. Optionally the filter comprises atemporal component in the case that filtering video data is required. Apixel-wise modulus operator is incorporated to remove the sign. Thenumerator of Equation 1 corresponds with discontinuity detection and thedenominator corresponds with envelop detection. Applying Equation 1 on asignal as depicted in FIG. 1A results in a signal as depicted in FIG.1C. An additional effect of the scaling unit 201 is that the depth D₀(x, y) of an original scene which is less than the depth range of thethree-dimensional image display apparatus will be increased. Thus anydepth map D₀ (x, y) is scaled such that the three-dimensional effect ismaximized given the capabilities of the image display apparatus.

FIG. 2C schematically shows an embodiment of a scaling unit 203comprising a clipping unit 216. Applying Equation 1 might result insuperceding the depth range of the three-dimensional image displayapparatus. This causes a maximum overall depth effect. For some types ofthree-dimensional image display apparatus this is helpful, e.g. for anautostereoscopic display with a lenticular screen. Whenever supercedingthe depth range is really not allowed a clipping post-processingoperation is performed by means of the clipping unit 216.

FIG. 3 schematically shows an embodiment of a scaling unit 300 based onluminance contrast detection. The scaling unit 300 comprises:

-   -   a luminance contrast detection unit 302 arranged to determine a        particular luminance contrast value of a particular pixel with a        neighboring pixel, with the particular pixel belonging to a        two-dimensional image 313 which is a view of the        three-dimensional model; and    -   luminance contrast dependent scaling means 304 arranged to scale        a depth value of an element which corresponds with the        particular pixel on basis of the particular luminance contrast        value.        The working of the scaling unit 300 will be explained by means        of an example. At the input connector 306 a two-dimensional        image 312 is provided, with each pixel having a luminance value.        The image 312 shows a white background. In front of the        background two objects are located: a white object 316 without        texture and a grey object 314. The distance between the white        object 316 and the background is bigger than the distance        between the grey object 314 and the background. This can be        observed by inspecting the depth profile 318 corresponding to a        row 313 of pixels of the image 312. This depth profile 318        comprises two blocks 319 and 321 which correspond to the white        object 316 and the grey object 314, respectively. At the input        connector 308 the depth map is provided to the scaling unit. The        depth map comprises a set of elements, with each element having        a value representing a depth value of the corresponding pixel of        the two-dimensional image 312. Because the white object 316        lacks texture and contrast with the background, this object 316        is hardly visible in the two-dimensional image 312. Hence, it is        not very useful to take the depth values 319 related to the        white object 316 into account when the depth map has to be        scaled to e.g. the capabilities of a three-dimensional display        apparatus. The opposite is true for the depth values 321 related        to the grey object 314. At the output connector 310 of the        scaling unit 300 the scaled depth map is provided. Depending on        the settings of the scaling unit 300 the regions in the original        depth map corresponding with hardly visible objects, e.g. white        object 316, can be fully removed. Depth profile 322 shows only        one block 327 which corresponds with the grey object 314 and no        other block. With other settings of the scaling unit 300 these        type of regions are not removed but adapted based on scaling        which is determined by regions in the depth map which are        visible, e.g. grey object 314. Scaling means 304 might be based        on a scaling unit 200, 201, 203 as described in connection with        FIG. 2A, FIG. 2B or FIG. 2C, respectively.

An embodiment of a scaling unit which is based on sensitivity to colorsubstantially resembles the embodiment of the scaling unit 300 asdescribed in connection with FIG. 3.

FIG. 4 schematically shows an embodiment of an image display apparatus400 comprising:

-   -   receiving means 402 for receiving a signal representing a        three-dimensional model;    -   a scaling unit 404 for scaling the three-dimensional model into        a scaled three-dimensional model in a dimension which is related        with depth; and    -   display means 406 for visualizing of a view of the scaled        three-dimensional model.        The signal can be received from a broadcaster or read from a        storage medium as DVD. Optionally the receiving means 402 are        arranged to convert the received information which is stored by        means of a first type of three-dimensional model into another        type of three-dimensional model. The scaling unit 404        corresponds to one of the scaling units as described in        connection with any of the FIGS. 2A, 2B, or 3. The image display        apparatus 400 can be of any of the types as listed above.

It should be noted that the above-mentioned embodiments illustraterather than limit the invention and that those skilled in the art willbe able to design alternative embodiments without departing from thescope of the appended claims. In the claims, any reference signs placedbetween parentheses shall not be constructed as limiting the claim. Theword ‘comprising’ does not exclude the presence of elements or steps notlisted in a claim. The word “a” or “an” preceding an element does notexclude the presence of a plurality of such elements. The invention canbe implemented by means of hardware comprising several distinct elementsand by means of a suitable programmed computer. In the scaling unitclaims enumerating several means, several of these means can be embodiedby one and the same item of hardware.

1. A method of scaling a three-dimensional model (100) into a scaledthree-dimensional model (108) in a dimension which corresponds to aviewing direction of a viewer towards the three-dimensional model (100),characterized in that scaling is based on properties of human visualperception of the viewer.
 2. A method as claimed in claim 1,characterized in that a first one of the properties of human visualperception is sensitivity to a discontinuity (109-113) in thethree-dimensional model (100) in a dimension which is related withdepth.
 3. A method as claimed in claim 1, characterized in that a secondone of the properties of human visual perception is sensitivity to adifference of luminance values between neighboring pixels of atwo-dimensional view (312) of the three-dimensional model (100).
 4. Amethod as claimed in claim 1, characterized in that a third one of theproperties of human visual perception is sensitivity to a difference ofcolor values between neighboring pixels of a two-dimensional view (312)of the three-dimensional model (100).
 5. A method as claimed in claim 2,characterized in that the method comprises a discontinuity detectionstep to detect a C₀-discontinuity (109-113) in the three-dimensionalmodel (100) in the dimension which is related with depth.
 6. A method asclaimed in claim 3, characterized in that the method comprises: aluminance contrast detection step to determine a particular luminancecontrast value of a particular pixel with a neighboring pixel, with theparticular pixel belonging to a two-dimensional image (312) which is aview of the three-dimensional model; and a luminance contrast dependentscaling step to scale a depth value of an element which corresponds withthe particular pixel on basis of the particular luminance contrastvalue.
 7. A method as claimed in claim 4, characterized in that themethod comprises: a color difference detection step to determine aparticular color difference value of a particular pixel with aneighboring pixel, with the particular pixel belonging to atwo-dimensional image (312) which is a view of the three-dimensionalmodel; and a color difference dependent scaling step to scale a depthvalue of an element which corresponds with the particular pixel on basisof the particular color difference value.
 8. A method as claimed inclaim 1, characterized in that the method comprises: a range detectionstep to estimate a range of depth values in a portion of thethree-dimensional model in the dimension which is related with depth;and a comparison step to compare the range of depth values with anoutput range of depth values.
 9. A scaling unit (200, 201, 203, 300) forscaling a three-dimensional model (100) into a scaled three-dimensionalmodel (108) in a dimension which corresponds to a viewing direction of aviewer towards the three-dimensional model, characterized in that thescaling unit (200, 201, 203, 300) is designed to scale on the basis ofproperties of human visual perception of the viewer.
 10. An imagedisplay apparatus (400) comprising: receiving means (402) for receivinga signal representing a three-dimensional model (100); a scaling unit(404) for scaling the three-dimensional model (100) into a scaledthree-dimensional model (108) in a dimension which corresponds to aviewing direction of a viewer towards the three-dimensional model; anddisplay means (406) for visualizing a view of the scaledthree-dimensional model (108), characterized in that the scaling unit(404) is designed to scale on the basis of properties of human visualperception of the viewer.